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All INSTRUMENT FOR THE EVALUATION OF 
INSTRUCTIONAL MATERIAL (FORM IV) 



Thi«s f v.vao «» ^ ^ s a, — 

w c»o a &C9|;^&ioc? d UdtfU LU CTX£' 

Ically select on the basis of some criteria soaterials which have 



been produced in great quantity, and to improve the utilization of 
these materials. Broadly conceived the data by this instrument comes 
to bear overarching questions: (What instructional materials shall 

be selected for purchase? (2) How can these selected instructional 
materials be effectively used in the educational program? 

In dealing with the issues of selection and use it was necessary 
to design an instrument that could be administered to a wide variety 
of material since the range of materials being produced has broadened 
proportionally to the quantity. As a result the central construct of 
the instrument were borrowed from those used to develop and analyze 
curriculum designs and are seen as necessary components of curriculum 
and instruction sequences. Under the four constructs: objectives, 

organization of material (scope and sequence), methodology and evalu- 
at ion, series of items of the ways these constructs are generally 
provided for in instructional materials was established. When instruct 
ional material does not meet any of the listed contingencies, space for 
an open ended response is provided. Following the systematic analysis 
of the material, a summary quantitative judgment is made for each 
construct and an overall judgment of the material as a learning package 
Also within the form are questions which elicit information on develop- 
ment and evaluation procedures used oy the producer in creating the 
learning package. 
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At this time the instrument has proceeded through four revisions 
resulting in considerable change in the items and the building of a 
glossary of special terms. In addition field testing after a training 
period of two to four hours had produced data on rater reliability in 
the neighborhood of eighty percent as they were gathered over a variety 

of instructional packages. These data lend credence to the consistency 
of the Instrument. 

Due to the lack of familiarity with the technical language and 
the design framework of the instrument, a training period is necessary 
for maximum utilization. In the taraining period using teams of two or 
three in assessing materials has proven to oe more productive than 
having individuals working alone. Moreover, the instrument serves a 
dual purpose as it trains teachers to systematically examine material, 
through sensitizing them to the producers intended use of the package. 
The maintaining of the integrity of the intended instructional design 
of new learning packages outside of the developmental setting has heen 
of major concern to authors and producers who have posited failure to 
do so as a major contrioutor to innovative learning pacUger. not per- 
forming to their original standard. It has also not been uncomnon in 
field trials of the instrument to witness the development of awareness 
among teachers that scarcely any Instructional material is a self con- 
tained learning package for a range of learners. Drawing attention to 
the way the Instructional materials satisfy the four constructs in its 
design, teachers relate the feasibility of the learning package to the 
needs of target populations of children. Usca ove:. time in examining 
Instructional materials for purchase, a school system could build a 
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valuable bank of data on strengths and weaknesses of competing 
materials . 

At this time Form IV of the instrument has not been used in 
predictive validity studies to determine if there is a carryover of 
its second objective into classroom Instruction. However, some of 
the research on instruction Indicates that systematic attention to 
instructional design constructs of stating objectives, selecting 
appropriate transaction, and evaluating results does have a positive 
influence on Instructional outcomes. Such evidence gives one con* 
fidence that a systematic appraisal of Instructional materials is of 
value to the producer, the teacher, and the student. 




BVALIJATIOM OF CURRICULUM MATERIALS (Form IV) 



. OBJECTIVES 



A. Arft th«r# object IvM sta-'ted for the use of the material? 
1* General objectivee? : 

Inatructlonal objectivee? 

3* Are the objectives ftated in behavioral terms ?^ 



Yea No 







Yes^ 


No 




„ No 


Yes_ 


No 



If stated in behavioral terms, 

St The type of behavior? 

Condition! under which it will appear? 
Level of performance expected? 



b. 

c* 



do the objectives specify: 



Yes. 

ms, 

Yes_ 



5* List examples of objectives, 



No, 

No, 

N0_ 



B. If there are no objectives stated for the use of the material, are 

the objectives instead implicit^ or readily obvious? ye s H o 

1. If yes, please outline below what objectives you believe xovern 
the purpose of the material. 

C. What appears to be the source of the objectives (both stated and implicit 
Objectives) ? 



1 . 



2 . 



3. 



4. 



Are the oojectives related to a larger frame of instruction? 

Yes_ No 



Are the objectives specific to a subject skill? 

Are the objectives related to a broader behavioral 
to be developed over a period of time? 

What seems to be the emphasis of the objectives: 
appropriate) 



Ye s N o 

pappern^ that is 
Yes N6^ 

(Check as many as 



a. Attitudlnal ^ 

b. Motor Skills __ 

c. Cognitive development skills^ 

d. Subject skills 



5. Are the objectives dram from: (check as many as appropriate) 

a. A learning approach^ 

b. Society needs (citizenship) 

c. Demands of the subject ’ 

a. Demands and needs of chiiy 



Page 2. 



0* Quantitative rating of objectives 

(Directions: Please maue andX on the rating scale below at Che point 

which represents your best judgenent on the following criteria. Please 
place the X on a specific point.) 



A — .1 ■ ■ I iiT 



S. 



Objectives * vague, 
unclear, or missing. 
Those Included not 
useful. Fails to dl&t 
distinguish between 
general and instruc* 
tional objeci'.ives, 
mixes various types 
of objectives, con- 
fusing to the teacher. 



Average, some of the crJ 
criteria for objectives 
met, some missing, at 
iagMeb inconsistent, ob- 
jectives only partially 
oi^erational for the 
classroom teacher. 



UsrXhexobjectives are 
stated clearly and 
in behavioral terms, 
doth general and 
instructional ob- 
jectives a^a stated 
in a consist: ent 
conceptual frame- 
work. Excellent, 
one of the best, 
useful for a teacher. 



B. 



Ill ORGANIZATION OF THE MATERIAL (SCOPE AND SEQUF.rTCE) 

A. Has a task analysis^ been made of the material and some relationship 
specified between the tasks? yes No 

If a task analysis has been made, what basis was used to organize the 
materials: (Check as many as appropriate) 

1. Errorless discrimination ^ 

2. Simple to complex 

3. Figure-ground^^ 

4. General to specific 
3 . Logical order ^ 

6. Chronology 



C. 



0. 



If no indication of a task analysis has been made, what assumptions 
do you believe the authors have made concerning the organization ofh 
the instructional sequence of the material? 

Is there a oasis for the scope of the material included in the instruct- 
ional package? jjo 



If there is a oasis, is it: 

a. Related to a subject area 

b. To a motor skill development 

c. To a cognitive skill area 

d. To an affective response system^l 
c/ Other (please specify) 



Yes. 

Yes. 

Yec, 

Yes* 



. No. 
. No. 
. Nol 



2 . 



Has the scope been subjected to analysis for; 
a. Appropriateness to students 
b# Relationship to other material 



Yes. 

Yes 



No. 

No_ 
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E. Is there a reconnended sequence? 

1. What is the basis of the reconanended sequence? (Chec as many 
as appropriate) 

a. Inter-relationships of a su )ject^^ _ 

b. Positive reinforceahnt and programed sequence^ ^ 

c. Open ended development of generalization^^ «««««, 

d. Advance organizers (cognitive) 

e. Other (please specify) . 



F. Briefly outline the scope and sequence. 



Q. Quantitative rating of organization of the materials (Scope and 
Sequence) 

(Directions; Please make an X bn the rating scale below at the point 
which represents your best judgement on the foiakowing criteria. 

Please place the X On a specific point.) 



Sequence Illogical 
or unstated, teach- 
is left to puzzle it 
out. Does not appear 
to have subjected 
materials to any 
analysis to build 
an instructional 
design. Scope is un- 
certain, seems to 
contradiet sequence. 
Little help uninten- 
tionally to teach- 
er or children in 
organizing material. 



Average in organi- 
zation. Sofne help 
.but teacher must 
supply much of or- 
ganizational se- 
quence. Scope some- 
what limited, nay 
be too narrow (or 
broad) . Sequence 
is not detailed c 
enough and may not 
have been tested 
with a range of 
children. 



Excellent organi- 
zation of scope and 
sequence . Con- 
ceptually developed 
based on a consistent 
theory tas: analysis 
or other appropriate 
Investigation has 
been done. Tested 
for appropriateness, 
of recommended 
sequence . 



III. PETHODOLOGY 

A. Does the author(s) and/dr material suggest any aethodologica|i ap- 
proach? Ye s K d 

3. Is the methodological approach if suggested, specific to the mode 
of transaction? Ye s N o 

1. Does the mode of transaction^^: (Check as many as appropriate) 

a. Rely upon teacher-centric method^^ (largely teacher directing) 

Yes No 

b. Rely upon pupil-centric method^^ (largely self-directing) 

Yes No- 
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c. Require active participation by thb students? 

Yes No 

d. Passive participation >y the students? Ye s No 

e. Combination of active and passive participation by the 

students? Ye s No 

f. Direct students' attention to method of learning as well 

as the learning product? Ye s No 

g. Provide for variation among student s-uses several approaches 

to method? Yes No 



C. Does the methcdologj;! require unusual skills o «tained through 
specific training? Yes N o 

1. How much deviation is permitted in methodology? 

Muc h Som e littl e 

2. Does the methodology require unusual s. ills obtained through 

specific training? Yes No 

3. Is there any statement on how methodology was tested* any 

experimental evidence? Ye s No 

4. If you have tried the recommended metholology, how successful 
did it seem for your students? 

Most succeede d Approximately % succeede d 

Few Succeede d 

a. Please provide a brief description of the students x^ho were 
successful and those who were not successful. 



b. What Variations on recommended methodology have you used? 



D. In a jfief statement desert e the recommended methodology. 



E. Quantitative rating of methodology 

(Directions: Please ma, e an X on the rating scale below at the 

point which represents your best judgement on the following criteria. 



Please place the X 

.,,1 »■■■» ■ i. ■ 

Very little help is 
given on methodology, 
or methodology is too 
abstract and complex 
for most Suudents and 
teachers . Methodology 
appears to be unrelat- 
ed to content and an 
afterthought in the 
learning package . Too 



a specific point.) 



Gives some help to 
the teacher, but 
xTOuld like more. 

Some students 
would e a »le to 
cope with suggest- 
ed methodology, but 
others not. Doesn't 
appear to have been 
widely field tested. 



^ 

Uses a variety of 
modes in the trans- 
actions. Does not 
chain a teacher to a 
mode without ^eason 
but provides assistance 
for different abilities. 
Describes the field 
test of the methodology. 
Teachers will find 
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active or pas- 
sive for most students. 
Teacher required 
to participate fully 
with too many students 
at every step. Doesn't 
have appropriate method- 
ology for variety of 
learning abilities 
among students 



Teacher has to work out 
variety for scftdents 
with special learning 
difficulties. 



ttefehodology easy to 
use and believe 
students will respond. 
Methodology is part 
of goals of instruct" 
ion and not just 
vehicle for content. 



IV. EVALOmON 

A. Are there recommended evaluation procedures for teachers and 
students in the instructional package? Ye s N o 

1* Vfhat do the evaluation procedures emphasize* 

(Check as many as appropriate) 

a. Cognitive s ills 

b. Subject Si ills 

c . Psychomotor skills^ 

d. Affective responses20 

2. Arc the evaluation procedures compatible with the objectives? 

Yes No 

3. Are dvaluaticn procedures developed for several different 

levels: (CheC; as many as appropriate) 

a. Iimnediate feedbacl^.; evaluation for the pupil 

b. Evaluation for a variety of the areas in #1 above and 
over a period of ti me 

c. Immediate feedback evaluation for the teacher 

d. Evaluation on a norm referent^l 

e. Evaluation on a criterion referent2ii^ 

f). Are the evaluation procedures contained in the package? 

Yes No 

C. Does the evaluation give attention to both product and process 

learnings? Yes No 

D. Is there Information on now evaluation procedures were tested 

and developed? ye s N o 

E. Briefly state what evaluation procedures are included if possible 
give examples. 



F. Quantitative rating evaluation 



er|c 



(Dirhetions: Please make an X on the rating scale below at the point 

which represents your bettt judgement on the following criteria. 

Please place the X on a specific point.) 



Haphazard in approach. 
Product and progess 
learnings either en- 
tirely ngglected or 
confused. Lists items, 
but poorly construct- 
ed, no evidence of 
testing of evaluation 
approach. Studehts 
receive no assistance 
through feedback. 

Falls to recognize 
and examine different 
types of learning 
where appropriate. 



Some examples given 
raage of evaluation 
limited. Samples 
given ut limited 
and sketchy. 

Teacher finds use- 
ful that whicn is 
given ut needs more 
examples. Evaluation 
is limited to product 
or process. Unsure 
on whether evaluation 
has ever been tested, 
but seems logical 
though limited in types 
of learning examined. 



Mhfty suggest ions and 
helps in evaluation 
for tne teacher. Has 
criterion reference 
procedures where ap- 
priate. Student 
obtains assistance in 
learning through 
feedback evaluation. 
Gives attention to 
several kinds of 
learning consistent 
with objectives of 
learriiag package 



COMMENT 

A, Draw up an overall statement of the streagats and weaknesses of 
the materifl as an instructional package. Prepare your statement 
if it were to be addressed to your fellow classroom teachers 
who are going to use it to ma> e a decision on these instructional 
materials. 

3. Quantitative rating overall assesment of material 



(Directions- Please place an X on the point in the rating scale waich 
best represents your overall judgement of these materials. Please place 
place the X on a specific point.) 



-*• — « Il l 

Poorly designed, con« 
ceptually weak, and 
inconsistent or hap- 
hazard design. Does 
not appear to have 
)een field tested; 
inaccurate assumptions 
about children who 
will be using material. 
Overpriced, under- 
developed a bad 
-argaiii. 



Has strengths and 
weaknesses , but most 
teachers would find 
satisfactory. On the 
balance comes out 
about average , , wou Id 
need considerable 
supplementary effort 
by teacher. A compro- 
mise on price and 
availability. 



- 

Excellent one of the 
best y comparison with 
other available mater- 
ial. Theoretically 
and conceptually 
strong and careful In- 
field tested. Shows 
Consistent instruction- 
al design. Would 
recomment highly, 
well worth the price. 
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k Glossary of Terms Used in this Instrument 



1, Objectives stated in behavioral terms - a word picture of the type of 
behavior or behavior product which one might expect when the objective 
is achieved. Objectives stated in behavioral terms will usually 

behavior, state the conditions under which it will appear, an t e 
level of performance expected, e.g., the child will be able to spell 
(type of behavior). In formal and Informal writing (condition under 
which it will appear) , 9» percent of the words in his written work 
(level of performance). 

2, Implicit objectives • an examination of the content will permit the 
reader to readily identify the objectives that the student should ac« 
complish, even If the producer has not stated them. If a filmstrip 
gives the sequential steps in solving arithmetic problems using long 
division, one would assume the implicit objective to be to teach the 
student the process of long division. 

3, Broader behavorial pattern - instructional materials frequently are 
geared to goals that Include complex behavior which Is to be developed 
over time. Example; voting behavior as a function of citizenship in- 
volves a broader behavorial pattern which chains together a complex or 
behaviors ranging from knowing the candidates and the issues, to being 
registered, and luiowing how to operate a voting machine. The instru^ 
tlonal material may be designed to contribute to a broader behavioral 
pattern rather than a simpler, more specific oehavlor. Even if the 
objective is geared to a single specific uehavior, there should be some 
relationship to a broader oehavloral pattern. 

4, ..'Attitudinal o»>jectives - objectives that are designed to develop feel- 

ings and predispositions to act in accordance with internalized values 
and beliefs. These may be listed as attitudes, values, interests, and 
appreciations. They may be fairly direct as to develop in each student 
an interest in listening to a newscast at least once a day, or more complex 
as the forming of an attitude of critically evaluating the news by invest- 
igating the source of reports. 

5, Cognitive development skIIIs - objectives which have cognitive develop- 
ment skills (thinking) as a oasis will usually emphasize thinking 
processes as their focus, such as understanding, discriminating, utiliz- 
ing, chaining, and evaluating as opposed to emphasizing specific subject 

products. 
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6. Objectives drawn from a learning approach • objectives may be dram 
utilizing approaches to learning, in some cases emphasizing wholeness 
of learnings prior to fragmenting into specifics for instruction. 

Example: the student will become familiar with the background of the 
12th and 13th century European interest in colonies and trade, prior to 
studying the specific explorations. The extreme of the above approach 
would be a small step by step sequencing of the material on Europe in 
the 12th and I3th century in which concepts on European interest in 
trade and colonies were fed to the student on a programmed basis, 
eventually leading through the various explorations. These contrasting 
Objectives are based on different approaches to learning. 

7. Objectives based on demands and needs of child • objectives using this 
emphasis usually have as their focus some developmental sequence 
(physical, emotional or social) as their central organizer . 

Example: the student will express affection as well as receive 

affection. The behavior of expressing affection is developnentally 
more advanced than simply receiving affection. 

Example: the student will cooperate with another student on taking 

turns in using a game. If this o ^jective Is to oe taught, it is 
usually sequenced with other Objectives according to the way most 
chlldrei^ develop. 

u. Taslv analysis * the materials have .leen developed into specific tasks 
for tuc learner which have behavorial requirements that suggest a 
sequence for presentation and which allow an Oi>server to determine 
if the learner accomplishes the task. 

'•j • Errorless discrimination • the tasks are sequenced in such a manner 
that the student should move from step to step xvdthout mai^-ing errors. 

This technique is used in some types of programmed instruction. 

10. Figure-ground - the organization of materials, frequently perceptual 
in nature, in a field so that one stands out in a distinct way (figure) 
and the rest remains in the jacKground (ground) . Figure-ground organ- 
ization can be used with other characteristics such as sounds, where 
one dound is heard over and a >ove a background of others. 

11. To an affective response system - where recognition Is given to different 
levels of attitudes, from the simplest, of merely attending to an object, 
to the uilding up of complex attitudes which predispose one's behavior 
toward a wide range of stimuli, e.g. enjoying a variety of forms of music. 

12. Interrelationships of a subject - where the sui<ject matter contains a 

logical relationship of concepts and processes. Example: adding must 

‘ be mastered prior to multiplying. The local community is studied prior 
to more distant entitles of state or federal governmer.t. 

13. Positive reinforcement and prograviM^d sequence - where the material has 
been developed into small steps thii* the learner toward a larger 
concept through a sequence that patinits the learner to receive frequent 
reinforcemeat turouga unowledge of fight answers. 
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14. Open eaded development of generalization • the instructional sequence 
is purposely quite unstructured, e.g., letting the learner try out many 
possioilities and alternatives before arriving at a generalization. 

1 C vera • s framework of key vonccpts, crucial 

concepts »£ the larger body o£ .aaterlal. 
are strategically placed in the sequence, forming an ideational ladder 
to ^Ich other material can readily be related. In some materials a 
short summary preceding the main oody of instructional material delineates 
the key concepts or stresses their relationship to other concepts known 
by the learner, thus serving as advance organizers through the ideational 
anchors it gives to the learner for organizing, relating and remembering 
the new material. 

16. Mode of transaction • a transaction is the interaction of a learner and 
stimuli, in this context consisting of instructional materials. A mode 
is the channel that is used. Is the student asked to passively view, 
manipulate, verbally organize? Is the teacher an important part of the. 
mode through exercising control over the learner's channels of transaction? 
Is the student free to seek out channels of transaction or are they chosen 
for him? These are questions which must be answered when setting up modes 
of transaction (methodologies) to be used wdch instructional materials. 

17. Teacher-centric method - the teacher is largely responsible for choosing 
and directing the mode of transaction for the learner. Teacher-centric 
inodes of transaction usually prescribe that the ’’teacher will..." and are 
predicated on obtaining specific learner responses. 

18. Pupil-centric method - the learner is responsible for choosing the modes 
of transaction with the instructional material and is frequently left to 
evaluate and revise his behavior toward materials without teacher supervis- 
ion. 

19. Psychomotor skills - muscular or motor skills which require manipulation 

of material or objects. The ability to stack blocks is a psychomotor skill. 

20. Affective response - responses which emphasize feelings, emotion or 
degrees of acceptance or rejection stemming from internal attitudlnal 
lets* Such responses may be labelled attitudes, biases, interests, etc. 

21. Norm referent evaluation - judging a learner's performanr e by what other 
knowii groups of learners do on the same tasks. Achievement test scores, 
aptitude tests, and mental test scores report their results in norm 
refereut terms. The statement, "This particular learner scored at Ath 
grade level," is using a norm refereut evaluation of the learner's 
performance. 
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:22-i'Cri|.perion referent evaluation - the learner is judged on his ability 
to do k specified tasu or demonstrate the behavior appropriate to the 
jtask. The learner is judged on whether he can or cannot demonstrate the 
appropriate behavior that signifies task accomplishment and ^ not? judged 
bv compdrison of his performance with another group of learners. 
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